Make InferenceGateway an AI gateway - #425

The fleet gateway was a generic HTTP router that understood nothing about
the requests it forwarded: a ModelService got a URL, a caller reached one
by path having already worked out what the engine called itself, nothing
authenticated or counted tokens, and the gateway was a cluster-scoped
singleton on the control plane, so every prompt crossed whatever region
that happened to be.

These are the APIs for the gateway the design describes: one that speaks
the inference protocol, resolves the model a request names, and translates
the request for the endpoint that serves it. InferenceGateway stops being
a singleton and names the InferenceCluster it runs on, gaining a hostname,
TLS, caller-key auth, and a serviceSelector that scopes which services it
serves. ModelEndpoint stops being a URL and becomes a backend description.
ModelService gains a priority order, separating failover from deliberate
traffic shifting, and loses its address, since a caller now names it as
the model at any gateway serving it.

An endpoint's origin must be a hostname, not an address: Envoy AI Gateway
applies per-backend model rewriting, credentials and priority failover
only when every backend in a route is addressed by hostname, and silently
stops applying all three otherwise. So InferenceCluster gains
spec.gateway.hostname. The example manifests and the validator that checks
them against the generated models move with the APIs; the composition
functions still expect the old fields and are updated next.

Signed-off-by: Nic Cope <nicc@rk0n.org>

compose-inference-gateway installed Traefik, MetalLB and a megabyte of
vendored Gateway API CRDs on the control plane, because nothing else on a
fresh control plane provided them, and composed a single Gateway that
ModelServices hung path-prefixed HTTPRoutes off.

The gateway now runs on the InferenceCluster the XR names, which already
runs Envoy Gateway and Envoy AI Gateway, so this function installs
nothing: it composes the gateway-scoped objects onto that cluster through
its ClusterProviderConfig, and the control plane stops owning a gateway or
the Gateway API CRDs. It owns the Gateway and its listeners, an EnvoyProxy
carrying the usage-record access log, a SecurityPolicy authenticating
callers, a BackendTrafficPolicy that makes priorities mean something, and
a /healthz route with its own allow-all policy. The per-service routes and
backends come from compose-model-service.

Composing nothing is a valid outcome, and the conditions say why. Three
things are load-bearing and were only found running it: a cluster-scoped
composite doesn't default its composed objects' namespace, so they need it
set explicitly; ndots:5 makes the proxy stall resolving every short
backend hostname through the search list, so its dnsConfig sets ndots:1;
and without retry.numAttemptsPerPriority the priority failover silently
never happens.

Signed-off-by: Nic Cope <nicc@rk0n.org>

compose-model-endpoint parsed spec.url and composed a selectorless Service
plus a hand-written EndpointSlice on the control plane, with two
Traefik-specific workarounds baked in. With gateways plural, the objects a
gateway needs to reach an endpoint are per-gateway, and an endpoint can't
know which gateways serve it without reading the ModelServices that select
it. So compose-model-service composes them and this function composes
nothing.

What's worth keeping is deciding whether an endpoint is usable, which
belongs here rather than in every route that references it. An endpoint
naming a credential Secret that doesn't exist, or one without the key it
names, would otherwise be composed into each gateway's route and fail at
request time. EndpointReady puts that on the endpoint, so
compose-model-service can leave a broken endpoint out of a route. Whether
a backend actually answers is a question only a request can settle, so the
gateway's outlier detection handles that.

Signed-off-by: Nic Cope <nicc@rk0n.org>

compose-model-service composed one HTTPRoute on the control plane's
singleton gateway, fanning over backendRefs that each carried a URLRewrite
filter, which was the only reason Modelplane ran Traefik.

A ModelService now becomes an AIGatewayRoute matching the model a caller
names, plus, per endpoint, the objects a gateway needs to reach it and
translate for it: a Backend, an AIServiceBackend, and where the endpoint
has a credential a BackendSecurityPolicy and propagated Secret. Which
gateways serve a service is the gateway's choice: this reads every
InferenceGateway and keeps those whose serviceSelector matches, which is
what makes residency fall out of labels rather than needing a feature. The
objects are named per service, so two services selecting one endpoint
compose their own copies rather than fighting over one.

Three details are load-bearing and each has a test that fails when it's
undone: every backend is addressed by hostname, or Envoy stops applying
the model rewrite, host rewrite and credential while still passing
traffic; llmRequestCosts is what makes a streamed response report tokens;
and a caller header is stripped from an endpoint Modelplane doesn't
operate, ours kept. Weights compete only within a priority tier,
and the schema requires at least 1, since a weight of 0 drops a backend
from the load assignment rather than parking it.

Signed-off-by: Nic Cope <nicc@rk0n.org>

…rves

A composed ModelEndpoint pointed at its cluster gateway's IP and carried a
rewritePath for the control-plane gateway. Neither survives the reshape.
An address doesn't work: Envoy AI Gateway emits a STRICT_DNS cluster for a
route whose backends are all hostnames, which carries the per-priority
failover, and an EDS cluster for an address, where model rewriting, host
rewriting and the credential all stop applying while traffic keeps
flowing. So an endpoint's origin is its cluster gateway's hostname, and a
cluster is schedulable only once spec.gateway.hostname is set. Publishing
DNS for a cluster gateway becomes a prerequisite for placing models on it.

A caller names a ModelService but an engine only answers to the name it
was started with, so Modelplane injects MODELPLANE_SERVED_MODEL_NAME into
every engine container and expects args to reference it. The composed
endpoint carries the same value as spec.model, so a service rewrites one
name for a whole deployment. A user override of the same name is dropped
rather than honoured, since it would let the engine answer to a name
nothing routes to. The convention is documented on the args field; a CEL
rule enforcing it exceeds the schema's cost budget.

This also finishes endpoint residency. An InferenceCluster gains
spec.placement.metadata.labels, stamped onto every ModelReplica and
ModelEndpoint composed there. A ModelService selects endpoints by label,
so without it a region-scoped service couldn't select its own replicas,
and nobody could label them by hand because Modelplane owns them. Cluster
labels beat the deployment's template on a collision, since the cluster is
the authority on where it is.

Signed-off-by: Nic Cope <nicc@rk0n.org>

The e2e asserted a 200 from a path-addressed ModelService on the control
plane, which no longer exists. It now drives the gateway the way a caller
does and asserts the mechanisms the design turns on. The InferenceGateway
runs on the workload cluster beside the models, so the control plane
installs no gateway; two clusters are kept as the realistic topology.

The mock engine now behaves like a real one: it takes
--served-model-name=$(MODELPLANE_SERVED_MODEL_NAME) and 404s anything
else, so a 200 through the gateway proves the caller's ModelService name
was rewritten to the deployment's. It also reads a chunked request body,
since the gateway's ext_proc rewrites the body and Envoy forwards it
without a content-length. A wired-in DNS step stands in for the DNS a
platform publishes per cluster gateway, which an InferenceGateway needs to
address a cluster by name.

Beyond a 200 it asserts that an unclaimed model routes nowhere, that
/v1/models lists the service, that the Anthropic Messages API is
translated, and that the gateway emits a usage record with the engine's
token counts. Verify gates on the ModelService reporting RoutingReady, so
it doesn't curl while the engine is still rolling out.

Signed-off-by: Nic Cope <nicc@rk0n.org>

The hop from a fleet gateway to a cluster gateway crossed whatever network
separates two clusters as plain HTTP. The design calls for TLS in both
directions, with each cluster gateway holding a certificate, none of it
optional.

Where the CA lives is a decision the design leaves open and it shapes
everything. A composition function can't generate key material, and
cert-manager exists only on InferenceClusters, so a single shared CA would
mean one private key transiting every cluster. This diverges from "a CA",
singular: each cluster runs a CA for its gateway's serving certificate and
each gateway one for the client certificate it presents, both issued by
cert-manager on the cluster they belong to, so no private key leaves it. A
gateway publishes its client CA, an InferenceCluster collects every
gateway's into a ClientTrafficPolicy on its HTTPS listener, and
compose-model-service points each Backend at the target cluster's CA with
the gateway's client certificate attached.

The listener fails closed rather than open. The HTTPS listener replaces
the HTTP one rather than joining it, so port 80 can't serve the engines in
the clear; it is served only once there is a client CA to enforce with;
and a cluster publishes its hostname, which is what makes it schedulable,
only once it has also published its CA, so no endpoint exists before the
hop it describes can be mutually authenticated. Verified on a live
cluster: a client presenting no certificate is refused during the
handshake.

Signed-off-by: Nic Cope <nicc@rk0n.org>

A cluster gateway's serving CA and a fleet gateway's client CA are issued
by cert-manager on the cluster they belong to, and the certificate has to
reach the control plane so the other end can trust it. Reading it from
cert-manager's Secret through provider-kubernetes was insecure:
provider-kubernetes copies the whole observed object into status with
--sanitize-secrets defaulting off, so every CA private key was readable by
anyone who could get objects. That removed the whole basis for the
per-cluster CA design, whose point was that no key leaves its cluster.

trust-manager publishes each CA certificate into a ConfigMap, which the
control plane observes instead. A Bundle copies one named key out of a
Secret in-cluster and rejects any PEM that isn't a CERTIFICATE, so the
private key never crosses a trust boundary and can't be made to by
misconfiguration. With nothing reading Secret data back, --sanitize-secrets
is now safe to turn on as well.

Three constraints decide how trust-manager installs. Its own webhook
Certificate races cert-manager's admission, so it waits for cert-manager
to report Ready; its default public-CA trust package blocks
the pod on an image pull no Bundle here needs, so it's disabled; and it's
gated on the self-signed Issuer, composed on every cluster, rather than a
CA that fleet-only clusters never compose. Both charts keep their CRDs on
uninstall, since provider-kubernetes can't observe an object whose CRD has
gone. The e2e now asserts an uncertified caller is refused mid-handshake
and that nothing answers on port 80, so a silently disabled mTLS policy
fails it.

Signed-off-by: Nic Cope <nicc@rk0n.org>

The manifests under docs/manifests were reshaped with the APIs, but the
prose around them wasn't, so the docs contradicted their own examples.
Every copy-pasteable request read ModelService.status.address, which no
longer exists, and built a URL from a namespace and service path, which
isn't how the gateway routes: a caller names the model and the gateway
rewrites it.

The request examples now read the base URL from the gateway that serves
the service and name the model. The platform gateway guide describes an
InferenceGateway that runs on a cluster and scopes itself with
serviceSelector, rather than a Traefik-and-MetalLB singleton, and getting
started's gateway section moved after registering the cluster, since a
gateway now needs one to run on. The
e2e README explains the two-cluster split as the shape Modelplane is for
rather than a CRD race. Also drops control-plane RBAC for resources that
no longer exist or now land on a workload cluster.

Signed-off-by: Nic Cope <nicc@rk0n.org>

A fleet gateway's CA and client certificate both put the InferenceGateway's
name in their commonName, and that name is a cluster-scoped resource name
with no short bound, so a long one produces a commonName cert-manager
rejects for exceeding 64 bytes. Both are now truncated to fit. The
commonName is cosmetic here: a cluster gateway trusts the CA by its
certificate and identifies a caller as a fleet gateway by that CA, not by
either name.

Signed-off-by: Nic Cope <nicc@rk0n.org>

The e2e stood in for per-cluster DNS by writing a CoreDNS hosts entry for
a hardcoded name and pointing the mTLS verify curls at it. Modelplane now
derives the name and composes the Service that resolves it, so the hosts
entry is gone and the curls read the derived name from the
InferenceCluster's status instead. The InferenceCluster manifest no longer
sets a gateway hostname.

Signed-off-by: Nic Cope <nicc@rk0n.org>